Explaining Synthesized Software

Motivated by NASA’s need for high-assurance software, NASA Ames' Amphion project has developed a generic program generation system based on deductive synthesis. Amphion has a number of advantages, such as the ability to develop a new synthesis system simply by writing a declarative domain theory. However, as a practical matter, the validation of the domain theory for such a system is problematic because the link between generated programs and the domain theory is complex. As a result, when generated programs do not behave as expected, it is difficult to isolate the cause, whether it be an incorrect problem specification or an error in the domain theory. This paper describes a tool we are developing that provides formal traceability between specifications and generated code for deductive synthesis systems. It is based on extensive instrumentation of the refutation-based theorem prover used to synthesize programs. It takes augmented proof structures and abstracts them to provide explanations of the relation between a specification, a domain theory, and synthesized code. In generating these explanations, the tool exploits the structure of Amphion domain theories, so the end user is not confronted with the intricacies of raw proof traces. This tool is crucial for the validation of domain theories as well as being important in every-day use of the code synthesis system. It plays an important role in validation because when generated programs exhibit incorrect behavior, it provides the links that can be traced to identify errors in specifications or domain theory. It plays an important role in the every-day use of the synthesis system by explaining to users what parts of a specification or of the domain theory contribute to what pieces of a generated program. Comments are inserted into the synthesized code that document these explanations. Introduction The Amphion project in the Automated Software Engineering group at NASA Ames Research Center is investigating technology to support the development of highassurance software. Amphion/NAIF [7, 5] is a domainspecific, high-assurance software synthesis system based on a specialization of the generic Amphion architecture. Amphion/NAIF takes an abstract specification of a problem in solar system observation geometry, such as “when will a signal sent from the Cassini spacecraft to Earth be blocked by the planet Saturn?”, and automatically synthesizes a Fortran program to solve it. Amphion greatly facilitates reuse of domain-oriented software libraries by enabling a user to state a problem in an abstract, domain-oriented vocabulary. The programs generated by Amphion/NAIF consist of assignment statements and calls to components from the SPICELIB software library in the NAIF toolkit. It takes significantly less time for an experienced user to develop a problem specification with Amphion than to manually generate and debug a program. More importantly, a novice user does not need to learn the details of the components in the library before using Amdomain-oriented vocabulary. The programs generated by Amphion/NAIF consist of assignment statements and calls to components from the SPICELIB software library in the NAIF toolkit. It takes significantly less time for an experienced user to develop a problem specification with Amphion than to manually generate and debug a program. More importantly, a novice user does not need to learn the details of the components in the library before using Amphion to create useful programs. This removes a significant barrier to the use of software libraries. Amphion uses deductive synthesis in which programs are synthesized as a byproduct of theorem proving from an application domain theory, such as the domain of solar system observation geometry. In this paradigm, problem specifications are of the form ∀ ∃ v v v v x y P x y [ ( , )], where v x and v y are vectors of variables. The theorem prover generates constructive proofs in which witnesses have been produced for each of the variables in v y . Amphion/NAIF demonstrates that, using deductive synthesis, it is possible to create domain-specific systems that enable users to generate highassurance software cost-effectively. Deductive synthesis has several potential advantages over competing synthesis technologies. The first is the wellknown but unrealized promise that developing a declarative domain theory costs less than developing a special-purpose synthesis engine through ad-hoc techniques. The second advantage is that synthesized programs are correct relative to a domain theory and the component library. The third advantage is that this relative correctness is rigorously documented in a verification proof, thereby potentially providing more understandable and readable code than even the best documented manually developed code. This latter potential advantage is considerable, as state-of-the-practice code generators produce programs that are unfit for human consumption or human maintenance. This potential of deductive synthesis has not been realized because raw, mechanically-generated proofs are also unfit for human consumption, and can only be understood through a laborious process by experts in theorem-proving technology. This paper describes a tool we have developed (and have nearly finished implementation as of May 1998) to provide explanations of programs generated by Amphion. It will first be used to automatically insert comments into generated code that document the relation between program variables and parts of the specification and domain theory. It will then be used to provide a dynamic web-browsable explanation of all aspects of generated code, enabling a user to probe the rationale for a generated program in terms of the domain theory. Based on our past experience in validating and debugging Amphion domain theories, this facility is expected to enable domain experts to home in on parts of a domain theory that lead to faulty programs. This is part of the larger goal of the Meta-Amphion project: enabling domain experts to construct, validate, and maintain their own high-assurance software synthesis systems. The next section of this paper introduces the explanation tool through a simple example from the NAIF domain. Section 3 then provides an overview of the Amphion deductive synthesis system, sufficient to understand the technical development in the rest of the paper, illustrated with this same example. Section 4 then illustrates the mechanics of generating an explanation. Section 5 develops the mathematical framework for generating explanations. Section 6 describes our tracing algorithms. Section 7 then relates this work to previous work and discusses future work. Introduction to the Explanation Tool We illustrate the need for an explanation tool with the following example. Figure 1 shows a simple specification given to the Amphion/NAIF system. Figure 1: A simple specification for Amphion/NAIF This specification depicts the constraints on a program that takes a time as input and produces as output the distance between the center of Earth and the center of Mars at that time. In general, specifications are given at an abstract level and programs are generated at a concrete level (input/output parameters are exceptions to this). Abstract objects are free from implementation details; thus, a point is an abstract concept, while a Fortran array of three real numbers is a concrete, implementation-level construct. The concrete array may represent the point, in which case this representation must be further specified by a coordinate system and the origin and orientation of the coordinate axes. The input in Figure 1(UTCIN in the upper left of the diagram) is depicted by a chevron with an arrow pointing into it. This chevron is connected to two other objects, Time-of-Interest and UTC-Calendar. Time-of-Interest is an abstract time, that is, an object representing the concept of a particular time. UTC-Calendar is a concrete time system. The arrows indicate that the abstract Time-of-Interest is represented in a program by a data object, UTCIN, which is interpreted as a time coordinate in UTC-Calendar format. Body-Earth and Body-Mars are abstract objects; each represents the state (space-time location and orientation) of its respective planet at the given time. Hence, Body-Earth is constrained to be the state of the planet with the name Earthtime, that is, an object representing the concept of a particular time. UTC-Calendar is a concrete time system. The arrows indicate that the abstract Time-of-Interest is represented in a program by a data object, UTCIN, which is interpreted as a time coordinate in UTC-Calendar format. Body-Earth and Body-Mars are abstract objects; each represents the state (space-time location and orientation) of its respective planet at the given time. Hence, Body-Earth is constrained to be the state of the planet with the name Earth at Time-of-Interest. The variable Distance-Earth-Mars is the abstract distance between the centers of the two bodies. Finally, the output EMDIST (the chevron in the lower right of the diagram) is the concrete representation of the abstract distance in kilometers.Amphion/NAIF will generate the program shown in Figure 2 from this specification: SUBROUTINE EARTH0 ( UTCIN, EMDIST ) IMPLICIT NONE DOUBLE PRECISION SLOC(3) C Code for EARTH-MARS-DISTANCE C Request-id: REQ-1998-04-06-14-58-47-423 C Parameters C MARSNA is Mars-NAIF-ID INTEGER MARSNA PARAMETER ( MARSNA = 499 ) C EARTHN is Earth-NAIF-ID INTEGER EARTHN PARAMETER ( EARTHN = 399 ) C Input variables CHARACTER*(*) UTCIN C Output variables DOUBLE PRECISION EMDIST C Functions DOUBLE PRECISION VDIST LOGICAL RETURN C Local variables DOUBLE PRECISION E DOUBLE PRECISION SMARS ( 6 ) 1 This version of the domain theory only has one representation for distances, namely kilometers. Thus this concrete object is not parameterized. 2 Amphion/NAIF also generates a driver main program for this subroutine, but in the interest of simplifying the presentation, this is not shown here. DOUBLE PRECISION SEARTH ( 6 ) DOUBLE PRECISION PSMARS ( 3 ) DOUBLE PRECISION PSEART ( 3 ) C Error handling IF ( RETURN() ) THEN RETURN ELSE CALL CHKIN ( 'EARTH0' ) END IF CALL UTC2ET(UTCIN,E) CALL SPKSSB(EARTHN,E,'J2000',SEARTH) CALL SPKSSB(MARSNA,E,'J2000',SMARS) CALL ST2POS(SEARTH,PSEART) CALL ST2POS(SMARS,PSMARS) EMDIST = VDIST(PSEART,PSMARS) CALL CHKOUT ( 'EARTH0' ) RETURN